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EXECUTIVE SUMMARY 



In response to the need to ensure that standards fulfill their promise of supporting a system of 
education that leaves no child behind, McREL created a multimedia, multi-site, ongoing 
nationwide dialogue related to standards-based reform efforts. These dialogues provided 
participants with opportunities to share underlying assumptions, beliefs, and research about how 
to effectively implement standards-based reform to realize the overarching goal of the standards 
movement — to help all students achieve high standards. 

The goals of this National Dialogue project included: 

• creating collaborative inquiry that involves the entire range of citizenry and seeks 
to offer solutions that represent the best thinking of a wide range of people across 
the nation; 

• fostering an exchange of information and ideas at all levels of the system and 
among educators and stakeholders from across the nation on issues related to 
standards-based education; and 

• helping all stakeholders create shared understanding about how to effectively 
implement standards-based reforms to realize the overarching goal of the 
standards movement — to help all students achieve high standards. 

To engage community members in this dialogue, McREL created a set of “conversation starter” 
materials designed to help communities engage in thoughtful deliberation on an issue of key 
concern for districts and schools in light of the No Child Left Behind (NCLB) Act of 2001 — 
how to help all students achieve high academic standards. 

This report details the outcomes of the National Dialogue over the course of the past four years 
and provides insights into how decision makers might capitalize on the findings to effect the 
changes necessary to ensure that all children succeed. This executive summary is intended to 
provide a brief overview of methods and findings, as well as the major implications of those 
findings. 

With respect to the overall outcomes of the conversations: 

• The public is interested in engaging in conversations about education, as 
evidenced by the level of interest in McREL’ s National Dialogue materials and 
process. 

• Dialogue participants agree that leaving no child behind would take much more 
than merely holding a school accountable for results and punishing failure with 
sanctions. 

• Conversation topics and emphases of concern have shifted over time — the 
public appears to be more attuned to the necessity of standards, though still 
struggles with fundamental issues about education (e.g., the puipose of 
education). 
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• Participants from different locations express slightly different perspectives on 
standards-based education. 

• Education accountability is important to the public; they consider it imperative 
that accountability be shared, that accountability not be construed as a single test 
score, and that accountability be accompanied by flexibility. 

• Participants consider resources necessary for ensuring student success, though 
differences among the public arc evident in how resources ought to be positioned. 

Anticipated outcomes of the National Dialogue were evaluated utilizing criteria that reflected the 
utility of the Dialogues (in terms of such factors as appropriateness, timeliness, and anticipated 
future use) and the benefits of the Dialogues (in terms of fostering shared understandings, 
fostering substantive knowledge on technical issues, fostering deeper appreciation of alternative 
perspectives, and in terms of encouraging broad engagement). The evaluation revealed that the 
National Dialogue met its intended goals: 

• Participants and moderators reported high levels of engagement and open, 
respectful conversations that were meaningful and relevant. 

• Demographically speaking, the conversations drew together diverse 
constituencies with varied perspectives, and although participants did not always 
reach consensus or completely understand one another’s perspectives, the 
exposure to varied understandings is crucial. 

• Participants did not report feeling disenfranchised because of technical issues — 
if anything, participants were deeply engaged in the ideas associated with 
standards-based education. 

• Although actions and outcomes at the community level directly emanating from 
Dialogue participation are not discernable, arguably the Dialogue served its 
purposes well — particularly in terms of engaging the public in the conversation 
and facihtating their discovery process in terms of their own and their fellow 
citizens’ perspectives regarding education. 

McREL’s experience with the National Dialogue project and the analysis of the data collected 
lead us to offer conclusions and recommendations for policymakers endeavoring to engage the 
full support of the public in improving our nation’s schools. Our recommendations address two 
areas — assessment and accountability and public engagement: 

• Current forms of accountability that focus on reporting test scores in individual 
content areas for groups of students at particular grade levels do not appeal - to 
have the same meaning and value for the public as they do for policymakers. 

• Parent participants expressed considerably more tolerance for variations in 
outcomes for their own children than policymakers would allow. 

• If policymakers want public education to be truly accountable to the public, they 
need to reframe accountability policies in ways that are meaningful to the public. 
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• The public wants multiple measures of performance and multiple ways for 
students to demonstrate their competencies. 

• Education leaders need to refine their communication to the public about 
accountability. 

• The current high-stakes testing accountability environment, though intended in 
part to increase public awareness and involvement in education, might actually 
cause the public to be further disengaged from the process, in part because of the 
technical nature of the conversations and because meaningfully involving parents 
in education might be superseded by attention to increasing test performance. 

• Education leaders need to reach out to the public — people want to be involved 
but do not know how to get involved. 

Our experience with the Dialogues suggests that the standards movement has reached a new stage 
in its evolution. The public no longer questions the wisdom of standards, assessments, and 
accountability per se , but rather, questions whether or not the focus on accountability actually 
helps students reach the most important goals in their development. It is not only academic 
success that is on the minds of parents and the public. Dialogue participants express deep concern 
about students’ civic-mindedness, sense of caring for others, flexibility and adaptability, work 
ethic, and creativity, as much as or, in some cases, more than, test scores. 

Ultimately, the conversations turn to fundamental questions about the purpose of our educational 
system, particularly in light of preparing students to cope with the uncertainties of the future. 
Participants express concern about the rapid pace of change and the new knowledge and skills 
students will need to acquire to thrive in the 21 st century. Discussion arises about the very nature 
of schooling — will it remain essentially as it is today, in buildings with children grouped into 
grades according to age, with one teacher to guide them? Or will the system become something 
few have imagined, much less discussed — one without the schools of today but perhaps with 
individualized learning plans for each student and a wide variety of ways (virtual and face-to- 
face) to reach goals and demonstrate proficiency? This is a discussion that should be nurtured in 
communities and is a question that McREL intends to pursue in our further work toward ensuring 
that educational leaders are prepared to meet the challenges ahead. 
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INTRODUCTION 



In response to the need to ensure that standards fulfill their promise of supporting a system of 
education that leaves no child behind, McREL created a multimedia, multi-site, ongoing 
nationwide dialogue related to standards-based reform efforts. These dialogues provided 
participants with opportunities to share underlying assumptions, beliefs, and research about how 
to effectively implement standards-based reform to realize the overarching goal of the standards 
movement — to help all students achieve high standards. 

McREL convened the initiating event for the Dialogues in Kansas City, MO in April 2001. This 
meeting generated interest in creating dialogues on standards-based reforms at the state and local 
levels. Over the next three years, similar dialogues and focus groups occurred in urban, suburban, 
and rural locations all around the country. The project, conducted under the auspices of the 
regional educational laboratory contract, was coordinated with work done for the Kettering 
Foundation to examine public perceptions of standards-based education, the implications of 
standards, and the meaning of accountability to the public. The findings from the Kettering study 
are incorporated into this report, which summarizes findings from the National Dialogue project 
and provides recommendations for policymakers. 

Goals of the National Dialogue Project 

The goals of the National Dialogue included: 

• creating collaborative inquiry that involves the entire range of citizenry and seeks 
to offer solutions that represent the best thinking of a wide range of people across 
the nation; 

• fostering an exchange of information and ideas at all levels of the system and 
among educators and stakeholders from across the nation on issues related to 
standards-based education; and 

• helping all stakeholders create shared understanding about how to effectively 
implement standards-based reforms to realize the overarching goal of the 
standards movement — to help all students achieve high standards. 

Toward these ends, the National Dialogue activities included developing, field-testing, and 
disseminating a collaborative inquiry process, gathering and disseminating insights from the 
Dialogues to foster a nationwide exchange of ideas, and providing opportunities for stakeholders 
to develop a shared understanding about standards. 

Development of the Dialogue Framework 

From the outset, McREL intended to start this nationwide initiative and follow it wherever 
conversations and interest took it. To facilitate this process, we explored a number of public 
engagement models that could be adapted for this purpose. The National Issues Forums 
( www.nifi.ors ) , a model of ongoing community dialogue sponsored by the Kettering Foundation 
(www. kettering.org) and carried out, in large part, by Public Agenda ( www.publicasenda.ors ) , 
appeared to provide an appropriate framework for our work. Thus, in 2002, we built a partnership 
with Public Agenda and created a set of Dialogue materials with the objective of situating the 
National Dialogue on Standards-Based Education reform within the ongoing efforts of the 
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National Issues Forums. By connecting the project with these existing efforts and tapping into the 
networks these organizations have created, it was possible to significantly expand and sustain the 
National Dialogue. 

In collaboration with Public Agenda, a set of “conversation starter” materials were designed to 
help communities engage in thoughtful deliberation on an issue of key concern for districts and 
schools in light of the No Child Left Behind (NCLB) Act of 2001 — how to help all students 
achieve high academic standards. The materials included a 1 2-minute video, produced both in 
English and in Spanish, explaining the conversation framework and showing clips from actual 
dialogues. A participant guide was created for use during the discussion. The guide also included 
participant survey instruments which were collected and analyzed following each dialogue. 
Finally, a guide for use by moderators in their training as well as during the facilitation of the 
Dialogue was created. This set of materials is included as attachments. 

Evaluation 

Beyond the development phase of the National Dialogue materials (including solicitation of 
feedback and appropriate adjustments where indicated), it was determined that the National 
Dialogue should be assessed relative to the extent to which the materials and the process itself 
met the following goals: 

• allowed for meaningful exchanges of information through conversation; 

• enabled various levels of stakeholders to be engaged in conversations; 

• provided a useful, appropriate, timely conversation with sustained benefits 
(participants anticipate using the information); 

• facilitated the development of a shared understanding of standards-based 
education; 

• facilitated the building of a bridge between community-level conversations and 
technical conversations; and 

• facilitated a better understanding of other stakeholder perspectives. 

Specific evaluation questions were derived from the objectives of the National Dialogue. 
Anticipated outcomes of the project were evaluated utilizing criteria that reflected the utility of 
the Dialogues (in terms of such factors as appropriateness, timeliness, and anticipated future use) 
and the benefits of the Dialogues (in terms of fostering shared understandings, substantive 
knowledge on technical issues, and deeper appreciation of alternative perspectives, and 
encouraging broad engagement). 

Evaluation data sources included site observations of dialogues, on-site participant surveys, and 
participant and moderator interviews and follow-up surveys. A detailed report of evaluation 
findings is included as Appendix A. A summary of the findings is included in the section of this 
report called Findings from the Dialogues. 
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Dissemination of Materials 



The materials were disseminated in a variety of ways, in addition to being made available for 
downloading or ordering through the National Dialogue website. Initially, a packet including both 
guides, the video, and an explanatory letter was sent to more than 100 national organizations with 
an interest in encouraging public engagement in education. These organizations were invited to 
disseminate the information to their own audiences and to contact McREL for technical 
assistance with conducting a dialogue. 

At the same time, McREL published an issue brief entitled Digging deeper: Where does the 
public stand on standards-based education? (Goodwin, 2003). This brief described the results of 
a series of focus groups conducted regarding public opinion on standards-based education and 
support for low-performing schools and invited readers to contact McREL for information about 
the National Dialogue. 

The brief was mailed or e-mailed to almost 59,000 individuals and organizations around the 
country. The response to the brief was overwhelming. National organizations such as the Public 
Education Network, the Institute for Educational Leadership, and the National Center for 
Learning Disabilities provided links to the brief on their own websites. State-level organizations 
such as the California State Library and the Electronic Library of Current Educational Research 
in Michigan did the same. 

The dissemination efforts resulted in nearly 60 individual and organizational requests for further 
information and technical assistance with conducting a community dialogue. Requests came from 
34 states and two Canadian provinces. In response to requests for support, McREL provided all 
materials needed for communities to host a dialogue and, in some cases, offered training for 
dialogue moderators and assistance with event planning. In return, hosting organizations were 
asked to allow us to collect data from participants and organizers regarding the impact of the 
Dialogue on their attitudes and actions related to standards-based education. 

Ultimately, McREL provided intensive support for 10 dialogues. Reports from each of these 
dialogues are posted on the website ( www.nationaldialosue.ors ) . A summary of the overall 
findings from these dialogues are described in this report. In addition, other local, state, and 
national organizations conducted dialogues on their own, using the conversation starter materials 
with minimal assistance from McREL. 

Although tracking web hits for the National Dialogue website only began in January 2004, 
290,672 hits have been tallied since that time representing 7,854 unique visitors. In addition, the 
site was visited by individuals in the United States and Guam as well as in 22 other countries. 

FRAMEWORK FOR THE DIALOGUE 

To provide some structure and an initial starting point for the conversation, the Dialogue on 
standards was based on a framework that proposed the following three approaches for helping all 
students achieve high standards: 
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Approach One: Help Students Achieve High Standards through 
Accountability 

The conversation starter materials provided to each participant describe this choice as follows: 

Choice One supporters say that for too long, we expected too little from our 
schools and our students — and got exactly what we expected. To change this 
situation, we need to define what students should know, test whether they have 
learned what they should know, and report the results. Doing so allows 
successful schools to receive the recognition they deserve and to act as models, 
while schools that fall short can be helped — and when necessary, pressured — to 
improve. And it keeps students who are not learning from falling through the 
cracks. In short, only by shining the bright light of accountability on the entire 
system will all schools be spurred to improve results for all students. 

Approach Two: Help Students Achieve High Standards by 
Providing Needed Resources and Support 

The conversation starter materials provided to each participant described this choice as follows: 

Choice Two supporters say that stanclards-based reforms should not only hold 
high expectations but also should provide high levels of support for all students, 
teachers, and education leaders to ensure that all students have the opportunities 
they need to succeed. However, in too many places, stanclards-based reforms are 
being sabotaged by what has been called the “evil twin” of standards — the use 
of high-stakes tests to shame and blame schools into improving without giving 
them the resources they need to improve. People who support this choice say that 
if we are really serious about setting high standards for all kids, we need to put 
our money where our mouths are. 

Approach Three: Help Students Achieve High Standards by 
Maintaining Flexibility and Local Control 

The conversation starter materials provided to each participant described this choice as follows: 

Choice Three supporters say that while standards are important guideposts for 
what kids should learn, we should avoid taking a one-size-fits-all approach to 
education and remember that schools are about more than just core or basic 
academics. People who support this choice say that all kids, parents, and 
communities are different. Therefore, we need to ensure that in putting standards 
in place, we don ’t “standardize ” schools so that they all wind up looking alike. 

People who support this choice say that parents want their schools to be 
accountable, yet not necessarily to lawmakers or school board members, but 
rather to them. 

OVERVIEW OF THE DIALOGUE PROCESS 

After viewing a conversation starter video, participants broke into small groups to deliberate each 

of these approaches, or choices. Led by a trained facilitator, they were asked to describe which 
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choice is closest to their own view and then to weigh the pros and cons of each choice. Following 
a discussion of the choice framework, each small group responded to the following questions: 

1. In our conversation about standards-based education, have we discovered 

any common ground? 

2. What were our important areas of disagreement — the things we have to 
keep talking about to work out our differences and move ahead? 

3. What arc the questions and concerns that need more attention? Are there 
things we need more information about? 

4. Flow can schools, parents, and the larger community do a better job of 
working together to make sure standards work as well as possible for all our 
students and that we are doing all we can to help them succeed? 

FINDINGS FROM THE DIALOGUES 

The objective in summarizing the findings across dialogues is to offer insights to policymakers 
regarding public 1 attitudes toward standards-based education and accountability and the effects of 
engaging the public in discussion about standards-based education. 

Attitudes toward Standards-Based Education 

Table 1 su mm arizes the key themes that emerged from 10 dialogues that occurred from 2001 
through 2004 in 10 different cities around the country. For these dialogues, McREL staff either 
facilitated or trained community moderators to facilitate the Dialogue, and observed the 
Dialogues and collected participant responses to surveys about the experience. 

General Observations 

A review of themes across the various dialogues reveals a consistent cry for balance in the current 
education reform movement. Participants pointed to strengths and weaknesses in each of the three 
proposed choices but, overall, found common ground in the notion that leaving no child behind 
will require pursuing a combination of all three. 

In order to be truly successful in leaving no child behind, participants agreed it would take much 
more than merely holding a school accountable for results and punishing failure with sanctions. 
The need for the community to share accountability for results and to ante up the resources 
needed to get results was often discussed. Participants increasingly rejected the “one-size-fits-all” 
notion embedded in the No Child Left Behind legislation and made clear the view that there is no 
silver bullet to closing the achievement gap. 

Changes Over Time 

McREL facilitators and observers noticed a change in the focus of the conversations over the 
course of the project period. In the first year, participants evidenced much more uneasiness with 



1 Of course we recognize that our National Dialogue community conversations have engaged only a small portion of 
the ‘public' and furthermore we are aware that our sample is not statistically representative of the larger ‘public’ for 
whom issues of standards and accountability are of interest. 




Table 1. Summary of Key Themes Organized 


by Dialogue 


June 2001 

Colorado (urban, suburban) 


October 2001 
New Mexico (rural) 


• Standards have had positive impacts for kids. 

• We need to revisit and return to our original 
purpose for standards. 

• CO’s reform efforts are driven by the 
statewide test. 

• The public needs to be engaged in 
meaningful discussion about reform. 

• Standards cannot address all of the public’s 
concerns about their schools. 


• Standards are written by experts who do 
not understand rural schools and 
communities. 

• Standards drive out locally-designed 
curriculum in rural schools. 

• Content standards are important but high- 
stakes tests cause concern. 

• The most valuable outcomes for students 
relate to becoming good community 
members (not reading and writing). 

• Support place-based education and show 
whether or not place-based education 
improves test scores. 


October 2002 

Missouri (urban, suburban, rural) 


December 2002 

Colorado (urban, suburban, rural) 


• Accountability is necessary, but needs to be 
fair and balanced. 

• More, and better, data is needed to guide 
reform. 

• Accountability has demanded results without 
granting flexibility. 

• Resources arc vitally important to meeting 
this challenge. 

• The whole community is responsible for 
leaving no child behind. 

• Substantial public support must be generated 
to leave no child behind. 


• Standards and assessments are important 
and necessary. 

• Accountability is important, but it 
shouldn’t be based solely on tests. 

• Accountability should go hand-in-hand 
with flexibility. 

• Schools need more resources to help all 
students learn. 

• Schools need better diagnostic data to 
guide decisions. 

• Current reforms arc a simple response to a 
complex problem. 

• School boards need to show their publics 
they are listening. 


April 2003 
Virginia (rural) 


May 2003 
Virginia (rural) 


• Accountability is important and necessary. 

• Accountability measures need to be re- 
examined. 

• Resources must be more targeted. 

• More community involvement is needed. 

• Local control is important, but should be 
balanced with external oversight. 


• Schools need more resources to achieve the 
goals of the No Child Left Behind Act. 

• Concerns about loss of local control are 
secondary to concerns about resources. 

• Accountability is necessary, but needs to 
be comprehensive and based on multiple 
measures. 

• Testing should not dominate the education 
process. 

• Teachers need to be held more 
accountable, but also supported with more 
training. 

• NCLB may be setting up schools for 
failure. 
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November 2003 
Connecticut (urban, suburban) 


January 2004 
South Dakota (rural) 


• Accountability is not just limited to schools 
and their employees; involvement of the 
other parties who are accountable needs to 
increase (including parents and community 
members). 

• A variety of assessments are necessary to 
determine students’ knowledge. 

• Although funding is not the answer, 
resources need to be appropriately 
positioned. 


• Accountability is not just limited to 
schools; involvement of parents, 
community members and students is 
crucial. 

• Standards are important for everyone, but 
there are different routes to reaching the 
standards. 

• Different types of assessments are 
necessary to determine students’ 
knowledge. 

• Although funding is not the answer, 
resources need to be positioned 
strategically. 


January 2004 
California (suburban) 


February 2004 
Michigan (urban, suburban) 


• Standards are “too standard;” they promote a 
“one-size-fits-all” approach. 

• Local control means letting the community 
define success and how to achieve it. 

• Accountability should be shared among 
educators, parents, students, community 
members, and politicians. 

• Resources arc critical in meeting students’ 
individual needs. 


• Accountability is important but it should be 
shared. 

• A variety of assessments will best 
determine what students know. 

• Sufficient resources are required to meet 
the needs of all students. 

• Educators should become politically active. 



the standards movement itself than in later years. In 2001. participants questioned the relevance 
of the standards to their own communities and the potentially damaging effects on students and 
schools of high-stakes testing. They also questioned whether or not standards-based reform could 
solve every problem of the public school system. 

In 2002 and beyond, following the signing into law of the No Child Left Behind Act. participants 
seemed to accept standards and accountability as an inevitable part of the public education 
landscape. Moreover, they supported the basic premise of the movement — to clearly specify 
what students should know and be able to do, measure their performance and publicly report the 
results — but questioned implementation details such as the quality of the measurement tools 
being used, the resources available to help students reach standards, and who should be held 
responsible if students don’t perform. 

Throughout 2003 and 2004, conversation participants evidenced greater knowledge and 
understanding of the standards movement, particularly as it is being played out through NCLB . 
The discussions were at a deeper level than previously and focused more on how to improve 
achievement, particularly in light of the great variation among students’ interests and abilities. 
Participants demonstrated a growing understanding of the enormity of the task and were quick to 
reject simple solutions to the problem of low student achievement. There appeared to be broad 
acceptance of the goal of leaving no child behind but they continued to express skepticism about 
the benefits of the accountability movement as currently constructed. 
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Differences Among Locales 

There were some differences among urban, suburban, and rural participants. Most notably, rural 
groups tended to favor preserving local control and to express concern that “outsiders” (such as 
federal or state governments) had too much say over what and how well their students should 
learn. All groups expressed concern that a single test score was not a sufficient, or even accurate, 
measure of student performance, but this concern was more pronounced among rural groups. The 
notion that all students should be prepared to go to college was also rejected by many participants 
across locales, but was emphasized more frequently in rural communities. 

Accountability 

Whereas most dialogue groups found strong common ground around the notion that 
accountability is a necessary component of improving schools, participants rejected the idea that 
accountability methods alone were sufficient to do the job. In addition, participants sought 
changes to the current accountability construct as it is embedded in NCLB. 

Accountability must be shared. Participants asserted that a broad range of stakeholders need to 
take responsibility for the success of a community’s students. It is not only unfair, but 
unproductive, to lay the entire blame for low achievement at the doorstep of the school. Many 
discussants pointed to the fact that students bear - no personal consequences for low performance 
on standardized tests (unless it is tied to promotion or graduation). Even teachers hold on to their 
jobs and continue to be paid at the same levels regardless of the performance of their students. 
And parents, who many feel are in the best position to either support or hinder their child’s 
academic success, face no consequences for failure and, in fact, may be “rewarded” by being able 
to move their child to a “better” school at the district’s expense. More people should be held 
directly accountable for the success or failure of students, participants said, although suggestions 
for accomplishing this were not forthcoming. 

Accountability should be based on more than just test scores. Throughout the conversations, 
there were repeated voices questioning the value of a single test score in determining whether or 
not children were learning. These comments usually were raised in the context of agreement with 
accountability principles (including the importance of assessment) along with concerns that a 
single test score from a single point in time does not give parents and the public the information 
they need to assess the quality of the education being provided. Yes, participants agreed, we do 
need to know whether or not our children arc learning, but do tests, as currently constructed, tell 
us all we need to know? Tests are viewed only as good as what they can measure and most 
participants questioned the ability of the current tests to measure all they want to know or believe 
is important about schooling. 

Flexibility should be provided in meeting accountability mandates. Participants expressed a 
concern that not only should there be multiple ways of demonstrating proficiency; there also 
should be multiple ways for students to attain proficiency. The notion embedded in the original 
standards movement — that what students need to know stays constant, but how they learn and 
how long it takes to learn varies by individual — has been lost in the current accountability 
movement, people said. The current system requires all students to take the same test on the same 
day, whether or not they have learned the material. It also requires all students to demonstrate 
their proficiency in the same way — by achieving a prescribed score on a standardized test. 

There was little support expressed for ascribing consequences (such as loss of funding or public 
labeling) to failing schools. Such policies were counterintuitive to most participants. Moreover, 
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participants repeatedly expressed concerns about the current trend toward narrowing the 
curriculum in order to meet test score goals. 

Resources 

Many participants entered into the conversation from the point of view that, if only there were 
more resources, the problem of low student achievement could be remedied. As the discussion 
continued, however, the general consensus across conversations was that more resources were 
necessary but not sufficient to solve the problem. Indeed, a more targeted, focused use of 
increased resources was deemed critical. Knowing what steps to take to improve achievement 
among those students who struggle most is what is needed. In the meantime, “throwing money at 
the problem” drew little support among participants. However, in several conversations, it was 
felt that more resources would provide an opportunity for more individualized instruction and that 
this individualized approach (which strategically aligns resources with needs) would be necessary 
to ensure that no child is left behind. 

Across the conversations it became evident that the public recognizes a relationship between 
education funding and student success, and that they even support the notion of increasing 
resources to raise academic achievement, provided the appropriated resources arc wisely 
expended. Pre- to post-participation survey data bear this out, with over 90 percent of respondents 
indicating a willingness to “redistribute resources or raise taxes” in order to help students achieve. 
This response was consistent both before and after participation in the conversation. 

Flexibility and Local Control 

For many participants, the issue of local control revolved around the role of the broader 
community in education reform. Few participants were willing to say that, in all cases, local 
control is best. Indeed, local control should be balanced with appropriate oversight to ensure that 
all students’ needs were being met, not just the ones who squeaked the loudest. But it was also 
noted that without any local control, the broader community loses its sense of responsibility for 
the outcomes and this is, on the whole, counterproductive to reaching the goals of school reform. 
The notion that it takes a village to raise a child was a common refrain. 

Similarly, participants wanted maximum flexibility in the way in which the accountability goals 
could be met. The one-size-fits-all approach to teaching and learning was consistently rejected in 
nearly every conversation. Two-thirds of survey respondents, both before and after the 
conversation, declared an interest in maintaining local control, even if such policies resulted in 
uneven quality across schools or diminished the “common school” experience of traditional 
public education. 

Effects of Public Engagement 

Evaluation data helped clarify the effects of participating in the Dialogue on attitudes and actions 
toward standards-based education and accountability as summarized below. 

Engaging in Meaningful Dialogue 

Participants uniformly agreed that they were able to actively engage in the conversation and, 
during site observations, participants expressed gratitude to have been included in the 
conversation about educational standards in their community. Moreover, during follow-up 
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interviews, moderators indicated that the participants were engaged. The following comment 
from a moderator provides a sense of the perceived level of participant engagement: 

They were highly engaged. Evidence: They stayed and talked at the end. They 
spoke to their peers about it, they sent us follow up e-mails and calls. They 
brought the topic up in dialogues that occurred after the session. 

During dialogue observations, through follow-up conversations and through the post-Dialogue 
questionnaires, participants agreed that the conversation was useful and meaningful. One follow- 
up comment captures the extent to which participants appreciated the opportunity to engage: 

I really enjoyed listening to the others and gained a new insight into the youth of 
today! The Dialogue is priceless and it helps me talk to the studen ts of today and 
make their education meaningful. 

Moderator perceptions of the utility of the conversation provide additional support for the 
participant’s own reflections, indicating that participants were pleased they were given the 
opportunity to engage in the topic and that they were able to parlay the National Dialogue 
conversation into other arenas. 

Follow-up responses from moderators also indicated that the conversations seemed relevant. For 
instance, an open-ended question regarding the relevance of the conversation included the 
following comment: 

Yes [it was relevant]. Participants were there because they were looking for a 
better understanding of standards-based education and to distinguish some of 
their concepts. 

Flowever, not all moderators agreed, as evidenced by the misgiving of one moderator, 
“[Participants] feel that a lot of talk happens and no action.” Such a sentiment was echoed in the 
conversations that we listened to during our site observations, with participants sometimes 
bemoaning the fact that their voices would be recorded and potential actions would be discussed, 
but in the end nothing would come of the conversation. 

By their nature, conversations on standards-based education and testing and assessment turn on 
very technical issues associated with benchmarks, alignment or the psychometric properties of 
testing instruments. Thus, the extent to which participants engaged with substantive ideas 
associated with standards-based education (e.g., defining standards, benchmarks, issues of 
assessment) in nontechnical ways was examined. 

During follow-up conversations and surveys, we asked participants to reflect on the extent to 
which they were able to understand the conversation, including any technical issues that arose. 
Participants indicated that they felt comfortable in their understanding, rating this item a 4.45 (on 
a scale from 1 to 5, where 1 indicates ‘not at all’ and 5 indicates ‘to a great extent’). In addition, 
moderators noted that they did not feel as though technical details about standards or testing 
“took over” the conversations. 

Appreciating Diverse Perspectives 

Dialogues were attended by a broad range of interested citizens, including local business leaders, 
parents, teachers, counselors, principals, retirees, and in many cases, students. Although the 



13 




majority of participants were White/Caucasian, this descriptive statistic must be accompanied 
with the caveat that one of our largest Dialogue sites in 2004 was in a rural area with very few 
community members and parents who were not White. Nevertheless, diversity of opinion was 
clearly evident, regardless of the demographic makeup of each dialogue group. In the words of 
one respondent to a follow-up survey, “It was wonderful to have the community speak and 
interact. Young and old.” 

In terms of whether participants felt that they had a better understanding of fellow citizens’ 
perspectives following the Dialogue, follow-up conversations and surveys revealed that 
participants agreed (though not strongly) that they learned about each others’ perspectives. 

Others’ perspectives can sometimes be unexpected — encounters with such different, diverse 
perspectives have the potential to engender self-questioning and self-reflection. Thus, during 
follow-up with participants, we asked whether there were any perspectives expressed during the 
Dialogue that surprised them. The following comments are indicative of the areas that provided 
new information or perspectives or that participants found particularly intriguing: 

One woman said that it was okay to teach to the test, but we have to agree what 
the test was. That was food for thought. 

1 was amazed that some educators can be numb to the social/emotional needs of 
our students. ..we cannot look at children as merely test passers... 

People teaching continuation alternative education , as well as our schools with 
high numbers of ELL students have incredible challenges which the system is not 
significantly addressing. They MUST have more support to begin meeting the 
standards with their populations. 

Moderators agreed that participants learned about each other’s viewpoints — and importantly, 
that participants learned about perspectives from those who are in different situations (ranging 
from different socioeconomic strata to different developmental periods as parents) from 
themselves. 

Developing a Shared Understanding of Standards-Based Education 

An outcome of significant interest was whether or not an individual’s attitudes and actions could 
be influenced by the experience of the Dialogue. Survey data revealed that participants had a high 
degree of interest in the topic of the Dialogue and ascribed significant importance to it. In 
addition, after being involved in the Dialogue, participants appeared to be slightly more 
concerned about a number of education issues, particularly those related to measurement and 
testing. 

Site surveys indicated that participants’ opinions about public policies regarding standards shifted 
following the conversation — from being less certain about what public policy on standards 
ought to be toward holding a more general sense or a definite sense of what those policies should 
be. 

During follow-up surveys, when we asked participants to indicate the extent to which they felt 
they had a strong understanding of standards-based education prior to participating in the 
conversation, participants rated themselves a 4.55 (on a scale from 1 to 5, where 1 indicates ‘not 
at all’ and 5 indicates ‘to a great extent’). It is important to bear - this in mind when one examines 
the mean rating for the extent to which participants felt they left the conversation with an 
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improved understanding of standards-based education. For this question, participants indicated 
that they did not feel that they learned that much (mean = 2.90; on a scale from 1 to 5, where 1 
indicates ‘not at all’ and 5 indicates ‘to a great extent’). Flowever, as one participant noted, “I 
think that most people do not understand what “standards-based” means. The Dialogue was 
useful in making it clearer to some what it means.” 

Although it is difficult to ascertain whether participants actually reached a shared understanding 
of standards-based education through their participation in the conversation, moderators 
suggested that alignment on some issues was evident — for some, this centered on the need for 
additional communication among schools, parents, and communities; for others shared 
understandings were around an agreed-upon vision for education priorities. Other issues — 
notably the role of standards and the concomitant and requisite actions to enable and enforce 
standards — failed to meet the benchmark of a shared understanding. Arguably, however, the role 
and goal of the National Dialogue project is not to enforce or even strive for consensus-building 
around such broad issues as what role standards ought to play in educating youth; rather the role 
and goal of the Dialogue is simply to bring to the fore the disparate issues, thereby ensuring that 
different voices are given an opportunity to be heard. In other words, reaching a shared 
understanding is different from reaching agreement on priorities and next steps. One moderator 
summed up this important sentiment by suggesting that “everyone’s opinion was heard and 
respected, but not necessarily agreed upon.” 

Moving Toward Action 

When moderators were asked whether the participants had engaged in any related activities 
following their participation, moderators suggested that some participants had followed up on 
how to open up and improve communication between parents and schools and that others had 
attempted to continue to engage their communities in dialogues and advisory groups. Importantly, 
one moderator noted that any actions associated with the National Dialogue — whatever these 
may be — would benefit from the “more objective approach to topics that include standards- 
based education” fomented through participation in the conversation. 

Follow-up conversations with participants, however, revealed that they are not always certain 
how they would utilize the information. Participants suggesting ‘actionable’ agendas often relate 
that they anticipate telling their neighbors and friends about the conversation and hope to urge 
others to get involved in similar - conversations in their community. Others indicated that they 
have become more committed to helping schools raise money in tax elections and some even 
expressed that they feel a greater sense of confidence in themselves as educators and parents. 

CONCLUSIONS AND RECOMMENDATIONS 

McREL’s experience with the National Dialogue project and the analysis of the data collected led 
us to a number of conclusions and recommendations for policymakers endeavoring to engage the 
full support of the public in improving our nation’s schools. These recommendations are 
presented in two parts — assessment and accountability and public engagement. 

Assessment and Accountability 

Current forms of accountability that focus on reporting test scores in individual content areas for 
groups of students at particular grade levels do not appear - to have the same meaning and value 
for the public as they do for policymakers. Participants in the conversations repeatedly rejected 
the notion that scores on such tests offered anything more than the broadest view of student 
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performance. Although measuring and reporting student progress was seen as important, high- 
stakes testing of cohorts of students, with results reported annually, was viewed as little more 
than a stalling point for determining how well a school is meeting its objectives. Such a testing 
and reporting scheme does not answer the questions parents truly care about when it comes to 
their own children’s performance or the performance of the schools their children attend. 

And yet, the public agrees that schools should be held accountable for meeting agreed-upon 
student performance goals that are based on a set of standards developed with input from the 
community. The question is not, should they be held accountable, but, rather, how should they be 
held accountable, for what, and to whom ? It would seem, based on our experience with the 
National Dialogue, that the public has a very different agenda for school reform than most 
educators and policymakers. 

Indeed, a recent Phi Delta Kappa/Gallup poll 2 (Rose & Gallup, 2003) shows standards to be very 
low on the public’s list of priority concerns about schools. Issues such as lack of funding, 
discipline, drug use, and overcrowding are ranked significantly higher as priorities than are 
standards or teacher quality. Dialogue participants, particularly those in rural communities, 
expressed concerns that the focus on reading and mathematics was causing schools to neglect 
such things as teaching students to be “good citizens... patriotic... and (to have) trust, civics, 
honesty, integrity.” Other groups mentioned critical thinking skills as an important paid of a good 
education; still others mentioned music, art, and simply learning to get along with others as 
opportunities that schools should provide. 

Parent participants expressed considerably more tolerance for variations in outcomes for their 
own children than policymakers would allow. That is, parents seemed comfortable with the 
notion that some children perform better academically than others. Rather than ensure that every 
child scored as well as every other child on a test, parents wanted to ensure that schools could 
support the development of every child’s unique talents and skills, whatever they were. There was 
concern expressed about the trend toward limiting options for noncollege-bound students and for 
those whose talents were less academic. 

If policymakers want public education to be truly accountable to the public, they need to reframe 
accountability policies in ways that are meaningful to the public. As more and more schools are 
labeled “in need of improvement” under No Child Left Behind, the label itself will become 
meaningless to parents who are likely to continue making decisions about which schools their 
children attend on a very different set of criteria than academic performance criteria such as 
safety, discipline, and proximity to home. The desire for the education system to focus on more 
than test scores is best summed up by this participant’s reflections on the Dialogue. 

We shared our obser\>ations of inconsistencies between schools, and 
acknowledged that beyond the standards, we must foster enjoyment of learning, 
social skills, and many concepts within the realm of Human In teraction. These 
are NOT given the time or the value we should be giving them, considering our 
goals for a society of involved, functional citizens and healthy families. The 
overwhelming number of expectations ; the moun tain of standards that is given 



2 Rose, L. C. & Gallup, A. M. (2003, September). The 35 th Annual Phi Delta Kappa/Gallup Poll of the Public’s 
Attitudes Toward the Public Schools, Phi Delta Kappan, 85. issue #. 41-56. 
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the highest priority versus fostering the whole person, is what the education 
system must look at. 

The public is looking for reassurance that schools are doing the job they expect them to do but 
annual standardized test scores do not give them that reassurance. They want multiple measures 
of performance and multiple ways for students to demonstrate their competencies. Policymakers 
should support research and development in the area of assessment in order to satisfy the need for 
a comprehensive, multifaceted assessment of student skills and knowledge that can still be used 
for accountability puiposes. In the meantime, the public is likely to continue to resist attaching 
too much importance to the publicly-reported scores on the tests. 

Education leaders also need to refine their communication to the public about accountability. 
Current accountability schemes, including No Child Left Behind, may have been oversold to the 
public, promising much more than they were meant to deliver. Policymakers need to make clear 
that measuring achievement is not, in itself, intended to improve achievement, but rather to focus 
attention on those students who need the most help. Moreover, if the policy community does not 
want education to be reduced to only “the three Rs,” it must work with the public to reform the 
system in ways that will allow individual students to develop their own potential while holding 
the adults in charge accountable in appropriate ways. 

Public Engagement 

Ironically, the current high-stakes testing accountability environment, though intended in paid to 
increase public awareness and involvement in education, might actually cause the public to be 
further disengaged from the process. There arc several reasons for this. 

First, the system of standards, assessment, and accountability is a highly technical one and 
difficult to interpret for the public. The general public does not understand the psychometrics 
associated with testing; they do not know the difference between “norm-referenced” and 
“criterion-referenced” tests and are not able to judge the quality of any particular assessment. 
Moreover, although states and districts have generally implemented a process that includes public 
input when developing standards, concepts such as “proficiency” and “benchmarks” can be 
confusing to parents who grew up with the bell curve. The technical nature of the process and 
discussions about it tend to intimidate and disenfranchise the lay public, further disengaging them 
from the process. 

Second, as schools focus more and more on achieving prescribed test results, there will be a 
natural tendency to devote more time to instruction, possibly neglecting parent involvement. It 
takes time to create and implement a robust, meaningful, parent engagement program at any 
school. As teachers focus more on reading and mathematics instruction in preparation for annual 
tests, the time to create structured opportunities for parent involvement is diminished. 

Finally, although we heard in the dialogues that accountability should be shared among parents, 
students, teachers, administrators, and the community at large, it was clear that participants did 
not know how to become involved in a deeper way. There is frequently an abundance of goodwill 
in communities toward their neighborhood school. There are people, with and without children in 
the schools, who are concerned and willing to help out, but they simply don’t know how to access 
the system and the system does not know how to incorporate them into it. Schools are a closed 
system, run according to an immutable format, often driven as much by the bus schedule as by 
actual educational or community needs. It is a system that is difficult to penetrate. Only the most 
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assertive, confident parent with plenty of time on his or her hands is likely to be able to 
participate in any meaningful way. 



Education leaders need to reach out to the public. They need to engage the public in dialogue 
about school reform and accountability, and respond to the concerns expressed, or run the risk of 
having their accountability policies become irrelevant to the people the policies are intended to 
reassure. Participants in the Dialogues were uniformly appreciative of the opportunity to discuss 
these issues with others and expressed the hope that those making policy would hear their 
concerns and respond. 



NEXT STEPS 

Our experience with the dialogues suggests that the standards movement has reached a new stage 
in its evolution. The public no longer questions the wisdom of standards, assessments, and 
accountability per se , but, rather, questions whether or not the focus on accountability actually 
helps students reach the most important goals in their development. It is not only academic 
success that is on the minds of parents and the public. Dialogue participants express deep concern 
about students' civic-mindedness, sense of caring for others, flexibility and adaptability, work 
ethic, and creativity, as much as or, in some cases, more than, test scores. Although most 
participants do believe that test scores provide some evidence of the success or failure of schools, 
this is not the only measurement of interest to them. In the dialogues we observed, there was a 
consensus that test scores do not tell the whole story. In fact, many participants noted, even if all 
students reach “proficiency” as defined by tests by the year 2014, they could not be certain that 
these students had learned and developed in all the ways that society needs and expects. 

Ultimately, the conversations turn to fundamental questions about the purpose of our educational 
system, particularly in light of preparing students to cope with the uncertainties of the future. 
Participants express concern about the rapid pace of change and the new knowledge and skills 
students will need to acquire to thrive in the 21 st century. Discussion arises about the very nature 
of schooling — will it remain essentially as it is today, in buildings with children grouped into 
grades according to age, with one teacher to guide them? Or will the system become something 
few have imagined, much less discussed — one without the schools of today but perhaps with 
individualized learning plans for each student and a wide variety of ways (virtual and face-to- 
face) to reach goals and demonstrate proficiency? This is a discussion that should be nurtured in 
communities and is a question that McREL intends to pursue in our further work toward ensuring 
that educational leaders are prepared to meet the challenges ahead. 

In 2005, McREL will move our National Dialogue work to a new frontier that focuses on the 
future of standards-based education 10 years from now. We will create scenarios for the future of 
education from a variety of perspectives and then share these with a national audience. 

Scenarios are not predictions of the future. Rather, they are plausible stories about the world in 
which we may live and work a decade or more from now. They highlight large-scale forces that 
could “push” the future in one direction or another. Scenarios about the future of education are a 
tool for educators in planning for the future of learning systems. They help leaders make 
decisions today that position them and their organizations to be most effective in the future. They 
force leaders to look beyond the immediate crisis of the day and to contemplate how one’s 
“official” view of the future could be misguided and cause us to make decisions today that are 
counterproductive to critical achievement goals for students today and in the future. 
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McREL intends to create further opportunities for dialogue among diverse stakeholders about the 
implications of the possible scenarios and options for actions by educational leaders in order to be 
prepared to meet the challenges ahead. In this way, the goals and objectives of the National 
Dialogue project can continue to be pursued. 
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APPENDIX A: EVALUATION OF THE NATIONAL DIALOGUE 



The goals of the National Dialogue include: 

• creating collaborative inquiry that involves the entire range of citizenry and seeks to offer 
solutions that represent the best thinking of a wide range of people across the nation; 

• fostering an exchange of information and ideas at all levels of the system and among 
educators and stakeholders from across the nation on issues related to standards-based 
education; and 

• helping all stakeholders create shared understanding about how to effectively implement 
standards-based reforms to realize the overarching goal of the standards movement — to 
help all students achieve high standards. 

Toward these ends, the National Dialogue activities have included developing, field-testing, and 
disseminating a Collaborative Inquiry Process, gathering and disseminating insights from the dialogues to 
foster a nationwide exchange of ideas, and providing opportunities for stakeholders to develop a shared 
understanding about standards. Beyond the development phase of the National Dialogue materials 
(including solicitation of feedback and appropriate adjustments where indicated), it was determined that 
the National Dialogue should be assessed relative to the extent to which the materials and the process 
itself met the following goals: 

• allowed for meaningful exchanges of information through conversation; 

• enabled various levels of stakeholders to be engaged in conversations; 

• provided a useful, appropriate, timely conversation with sustained benefits (participants 
anticipate using the information); 

• facilitated the development of a shared understanding of standards-based education; 

• facilitated the building of a bridge between community-level conversations and technical 
conversations; and 

• facilitated a better understanding of other stakeholder perspectives. 

Evaluation Questions 

Specific evaluation questions were derived from the objectives of the National Dialogue. Anticipated 
outcomes of the project were evaluated utilizing criteria that reflected the utility of the dialogues (in terms 
of such factors as appropriateness, timeliness, anticipated future use) and the benefits of the dialogues (in 
terms of fostering shared understandings, fostering substantive knowledge on technical issues and 
fostering deeper appreciation of alternative perspectives and in terms of encouraging broad engagement). 

These include: 

• Were participants engaged? 

• Was an open and respectful conversation maintained? 
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• Was the conversation meaningful? Was the conversation relevant (if so, in what ways)? 

• Were different constituencies involved in the Dialogue? 

• Has the conversation led to proposed actions (if appropriate)? 

• Do participants have a shared understanding of standards-based education [SBE] (where 
appropriate, have participants learned more about SBE)? 

• Did technical issues surface? Who was engaged? 

• To what extent are participants engaged with substantive ideas associated with SBE (e.g., 
defining standards, benchmarks, issues of assessment) in nontechnical ways? 

• To what extent do participants have a better understanding of fellow citizens’ 
perspectives? 

Table 1 details the relationship between overarching goals of the project and evaluation questions and 
data collection sources: 



Table 1. Relationship between National Dialogue Goal, Evaluation Questions, and Data Sources 



National Dialogue goal: 


Associated evaluation questions: 


Data Sources 


Allowed for meaningful 
exchanges of information through 
conversation 


Were participants engaged? 

Was an open and respectful 
conversation maintained? 

Was the conversation meaningful? 


• Site observations of 
dialogues 

• Participant and moderator 
interviews/follow-up 
surveys 


Enabled various levels of 
stakeholders to be engaged in 
conversations 


Were different constituencies 
involved in the Dialogue? 


• Site observations of 
dialogues 

• On-site participant surveys 


Provided a useful, appropriate, 
timely conversation with 
sustained benefits (participants 
anticipate using the information) 


Was the conversation relevant 
(if so, in what ways)? 

Has the conversation led to 
proposed actions (if appropriate)? 


• Participant and moderator 
interviews/follow-up 
surveys 

• On-site participant surveys 


Facilitated the development of a 
shared understanding of 
standards-based education 


Do participants have a shared 
understanding of SBE (where 
appropriate, have participants 
learned more about SBE)? 


• Site observations of 
dialogues 

• Participant and moderator 
interviews/follow-up 
surveys 

• On-site participant surveys 


Facilitated the building of a 
bridge between community-level 
conversations and technical 
conversations 


Did technical issues surface? Who 
was engaged? 

To what extent are participants 
engaged with substantive ideas 
associated with SBE (e.g., defining 
standards, benchmarks, issues of 
assessment) in non-technical ways? 


• Site observations of 
dialogues 

• Participant and moderator 
interviews/follow-up 
surveys 

• On-site participant surveys 
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National Dialogue goal: 


Associated evaluation questions: 


Data Sources 


Facilitated a better understanding 
of other stakeholder perspectives 


To what extent do participants have 
a better understanding of fellow 
citizens’ perspectives? 


• Site observations of 
dialogues 

• Participant and moderator 
interviews/follow-up 
surveys 

• On-site participant surveys 



A variety of data collection methods were used to examine the evaluation questions; 3 in some cases, more 
than one method was employed to address a given evaluation question in order to strengthen the 
credibility of the findings. Evaluation data included project documents and pre- and post-survey data 
(these surveys are included in the National Dialogue Community Conversation Framework: Leaving No 
Child Behind: How Can We Help All Kids Achieve High Academic Standards?). Four sites were 
identified as intensive study sites and as such, data collected included Dialogue observations, surveys, and 
follow-up conversations aimed at determining the extent to which each Dialogue goal was realized. The 
following section on Findings presents the data relevant for addressing each of the evaluation questions. 

Evaluation Findings 

Question: Did the Dialogue allow for meaningful exchanges of information through conversation? 

To address the first set of evaluation questions ( Were participants engaged? Was an open and respectful 
conversation maintained? Was the conversation meaningful?), site observation data and participant and 
moderator follow-up data were examined. 

Examining pre- to post-mean differences on the site surveys reveals slight increases on a number of items 
(see Table 2). Thus, after participating in the Dialogue, participants appeared to be slightly more 
concerned about a number of education issues. 4 Particularly high levels of concern arc apparent for items 
related to measurement and testing, with respondents indicating that the failure of test-driven reforms to 
address children’s needs and the use of test results to shame and blame schools among their highest 
concerns. 

When asked to identify other principles that should guide public education, survey respondents’ remarks 
align with those heard in the Dialogue. This was the case for written comments before the Dialogue and 
after the Dialogue. In particular, respondents noted that accountability should be shared with parents and 
students, that education needs to maintain some semblance of individuality - though it also should 
maintain and keep high, explicit expectations for all students, that a student’s home environment plays a 
role in their achievement, that resources are crucial, and that tests ought to be used in more diagnostic 
ways. Issues of equity were raised by a few respondents, but an overwhelming number of comments 
centered on finding ways to hold parents accountable for their child’s education and finding ways to 
individualize curriculum to meet all student needs. 



3 Because of the nature of the intervention, it was not possible or appropriate to design and evaluate outcomes of participants 
compared to outcomes of a nonparticipating comparison group. Therefore, the design of this evaluation study is a one-group, pre- 
post comparison of the effectiveness of the dialogue materials and the dialogue process and outcomes. 

4 The mean difference between pre- and post-ratings for the item “Schools lack incentives to help all students achieve high 
standards” is significant at p < .05. 
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Table 2. Participant Concerns on Education Issues (Pre- and Post Surveys) 



Question: How concerned are you about the following issues? 


Pre 


Post 


Mean 


SD 


Mean 


SD 


Test-driven reforms fail to focus on the needs of the whole child. 


2.77 


0.47 


2.82 


0.44 


Tests results are being used to shame and blame schools. 


2.77 


0.48 


2.81 


0.46 


Current accountability systems are taking away local control of schools. 


2.52 


0.63 


2.56 


0.58 


Schools lack the resources they need to help all children achieve standards. 


2.71 


0.52 


2.75 


0.51 


Schools have been following 'fads' instead of relying on what works. 


2.43 


0.68 


2.46 


0.67 


Schools lack incentives to help all students achieve high standards. 


2.16 


0.78 


2.29 


0.73 


Low expectations have let too many students fall through the cracks. 


n/a 


n/a 


2.39 


0.69 



Scale = 1 (not at all concerned) to 3 (concerned) 



During follow-up conversations and surveys, participants were asked to indicate the extent to which they 
agreed that all participants were given an opportunity to engage. The mean rating for this question was 5 5 . 
Thus, participants uniformly agreed that they were able to engage in the conversation. In addition, during 
site observations, participants expressed gratitude to have been included in the conversation about 
educational standards in their community. 

On the whole, moderators indicated that the participants were engaged. The following comments, 
extracted from follow-up moderator conversations provide a sense of the perceived level of participant 
engagement: 

[Participants were] very involved , everyone spoke, [but we] ran out of time. 

[Everyone was] fully engaged. Everyone was asking clarifying questions and giving 
thoughtful responses. 

They were highly engaged. Evidence: They stayed and talked at the end. They spoke to 
their peers about it, they sent us follow up e-mails and calls. They brought the topic up in 
dialogues that occurred after the session. 

Participants and moderators alike intimated that they were treated with respect and their voices were 
heard in the Dialogue. Thus, for instance, moderators suggested that because the facilitators were well 
trained, the conversational environment felt safe for participants. Moreover, one moderator noted that s/he 
heard “...many comments about the structure of the Dialogue providing safety and respect for their ideas.” 

During Dialogue observations, through follow-up conversations and through the post-Dialogue 
questionnaires, participants have noted that they find the conversations useful. Thus, when follow-up 
conversations and surveys queried the extent to which participants agreed that the conversation was 



5 For all follow-up survey data, a five-point Likert scale was used, where 1 indicated ‘not at all’ and 5 indicated ‘to a great 
extent.’ 
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relevant to important issues about schooling and education , the mean rating was a 4.70. This high rating 
is reinforced by the following comments regarding the relevance of the standards-based dialogue: 

Any conversation that focuses on what children actually experience in classrooms is 
relevant... 

We shared our observations of inconsistencies between schools, and acknowledged that 
beyond the standards, we must foster enjoyment of learning, social skills, and many 
concepts within the realm of Human Interaction. These are NOT given the time or the 
value we should be giving them, considering our goals for a society of involved, 
functional citizens and healthy families. The overwhelming number of expectations; the 
mountain of standards that is given the highest priority versus fostering the whole person, 
is what the education system must look at. 

We talked about the relevance of parent education at the earliest possible tune. Every 
penny given to parent education is a pound of prevention. 

The facilitator did a good job of encouraging everyone to participate (as did all of the 
participants) and kept us on task. I don't remember thinking that we were wasting time 
(as a matter of fact, we could have continued to discuss longer on most every item). 

In addition, participants indicated that the conversation was meaningful (mean rating of 4.0) and agreed 
that the conversation was useful (mean rating 3.56). One follow-up comment captures the extent to which 
participants appreciated the opportunity to engage: 

I really enjoyed listening to the others and gained a new insight into the youth of today! 

The dialogue is priceless and it helps me talk to the students of today and make their 
education meaningful. 

Moderator perceptions of the utility of the conversation (meaningful and useful) provide additional 
support for the participant’s own reflections, indicating that participants were pleased they were given the 
opportunity to engage in the topic and that they were able to parlay the National Dialogue conversation 
into other arenas: 

I think the conversation was a unique opportunity for most of these folks to talk about a 
fairly complex topic. 

[Pjeople found the conversation meaningful and useful. In the context of [our 
community] initiative, [participants] brought elements of this evening’s work in to later 
meetings. 

However, not all moderators agreed, as evidenced by the misgiving of one moderator, “[Participants] feel 
that a lot of talk happens and no action.” Such a sentiment was echoed in the conversations that we 
listened to during our site observations, with participants sometimes bemoaning the fact that their voices 
would be recorded and actions would be discussed, but in the end nothing would come of the 
conversation. 



A-5 




Question: Did the Dialogue enable various levels of stakeholders to be engaged in conversations? 



Based on demographic data from site surveys, we were able to determine that different constituencies 
were involved in the conversations ( Were different constituencies involved in the Dialogue?). Dialogues 
were attended by a broad range of interested citizens, including local business leaders, parents, teachers, 
counselors, principals, retirees, and in many cases, students. Although the majority of participants were 
White/Caucasian, this descriptive statistic must be accompanied with the caveat that one of our largest 
Dialogue sites in 2004 was in a rural area with very few community members and parents who were not 
White. Table 3 details the ethnicity of Dialogue participants. 

Parent respondents were also asked to indicate whether their children attended a private or a public 
school; not surprisingly the majority of parents (94.6%) indicated that their children attended public 
schools. Finally, in the words of one respondent to a follow-up survey, “it was wonderful to have the 
community speak and interact. Young and old” - a comment drawing attention to the diversity of 
perspectives represented in the conversation. 



Table 3. Ethnicity oi 


f participants 


Ethnicity 


N 


% 


African American 


46 


18.5 


Asian American 


2 


.8 


Flispanic 


7 


2.8 


Native American 


3 


1.2 


White 


187 


75.4 


Other 


3 


1.2 


Total 


248 





Question: Did the Dialogue provide a useful, appropriate, timely conversation with sustained benefits 
( participants anticipate using the information)? 

To address the third set of questions ( Was the conversation relevant [if so, in what ways]? and Has the 
conversation led to proposed actions, if appropriate?), data from participant and moderator interviews 
and follow-up surveys as well as on-site participant surveys were examined. 

Follow-up conversations with participants revealed that they arc not always certain how they will utilize 
the information - those suggesting actionable agendas often relate that they anticipate telling their 
neighbors and friends about the conversation and hope to urge others to get involved in similar 
conversations in their community. Others did indicate that they have become more committed to helping 
schools raise money in tax elections and some even expressed that they feel a greater sense of confidence 
in themselves as educators and parents. For many, engaging in the Dialogue is not an unfamiliar activity 
for them - the people who voluntarily attend such meetings arc the people who arc already engaged in the 
education system. 

Follow-up responses from moderators indicated that the conversations seemed relevant. For instance, an 
open-ended question regarding the relevance of the conversation included the following comments: 

[The conversation was] very relevant. Everything [parents] hear is Standards these days 
and they don't know what that means for their children. 
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Yes, I don't think most of them would've have come otherwise. 



Yes. Participants were there because they were looking for a better understanding of 
standards based education and to distinguish some of their concepts. 

Although many of the proposed actions remained at a very general level, when we asked moderators 
whether the participants had engaged in any related activities following their participation, moderators 
suggested that some participants had followed up on how to open up and improve communication 
between parents and schools and that others had attempted to continue to engage their communities in 
dialogues and advisory groups. Importantly, one moderator noted that any actions associated with the 
National Dialogue - whatever these may be - would benefit from the “more objective approach to topics 
that include standards based education” fomented through participation in the conversation. 

Question: Did the Dialogue facilitate the development of a shared understanding ofSBE? 

To address questions regarding shared understandings (Do participants have a shared understanding of 
SBE? Where appropriate, have participants learned more about SBE?), we examined data from site 
observations of Dialogues, participant and moderator follow-up interviews and follow-up surveys, and 
on-site participant surveys. 

Based on overall mean ratings, on-site surveys revealed that, prior to the conversation, National Dialogue 
participants felt the strongest about issues associated with using test results in a diagnostic fashion (rather 
than a shaming of schools and students). On the other hand, participants considered the tying of 
consequences to test results as an incentive for improvement the least important of educational principles. 
While the latter of these principles remained the least important for participants following their 
engagement in the National Dialogue process, the highest rated principle (based on overall means) 
following the conversation was that all necessary resources should be provided to help students meet the 
standards. Such shifts are perhaps indicative that the conversations have led people to reflect on or 
reconsider their initial positions regarding a set of principles that have been proposed to help all students 
achieve high academic standards. Table 4 provides means and standard deviations for all items associated 
with principles for helping all students achieve high academic standards. 



Table 4. Importance of Principles to Help Students Achieve 


High Academic Standards 


How important is each of the following principles to you: 


Pre 


Post 


Mean 


SD 


Mean 


SD 


We should provide all necessary resources to help students 
meet standards. 


2.87 


0.35 


2.86 


0.35 


Standards should not become a “one-size-fits-all” school 
curricula. 


2.69 


0.54 


2.66 


0.58 


Local districts and communities should have flexibility in 
setting and meeting standards. 


2.54 


0.61 


2.41 


0.61 


Test results should identify areas of need, not shame schools 
or students. 


2.91 


0.29 


2.82 


0.42 


Consequences should be tied to tests to give schools incentives 
to improve. 


1.63 


0.69 


1.78 


0.72 


We must accept no excuses for students failing to achieve high 
standards. 


2.01 


0.71 


2.03 


0.72 



Scale = 1 (not at all important) to 3 (very important) 
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Site surveys indicated that participants’ opinions about public policies regarding standards shifted 
following the conversation - from being less certain about what public policy on standards ought to be 
toward holding a more general sense or a definite sense of what those policies should be. Table 5 provides 
the percentage of respondents selecting each of the statements to describe their perspectives on standards 
policies both before and after participating in the conversation. 



Table 5. Pre and Post Participant Perspectives on Standards Policies 



Which statement best describes how you feel about our public policy 
to help all students achieve high standards? 


% 

Pre 


% 

Post 


I am not at all certain what our public policy should be. 


22.20 


14.30 


I have a general sense of what our public policy should be. 


50.00 


54.40 


I have a definite opinion of what our public policy should be. 


27.80 


31.30 



In a similar vein, participants were asked to reflect on three approaches to helping all children achieve 
high academic standards both before and after the Dialogue. These three approaches are aligned with 
those used to frame the National Dialogue conversations. Interestingly, the number of individuals who 
suggested they were unsure about the different approaches dropped for the first two options but remained 
constant for the third. An overwhelming number of Dialogue participants favor — both before and after 
participating — increased resources to help schools and students achieve. Table 6 provides these data. 



Table 6. Pre- and post- Reflections on Policies for Achieving High Academic Standards 



How do you feel about these approaches to 
making policy on helping all students achieve 
high academic standards: 


Pre 


Post 


% Favor 


% Oppose 


# Not sure 


% Favor 


% Oppose 


# Not sure 


We should focus on results & achievement, even 
if it means schools must narrow their curricula or 
lose some degree of local control. 


29.60 


70.40 


71 


32.50 


67.50 


26 


We should provide schools w/ adequate support 
to help all students achieve standards, even if it 
means redistributing resources or raising taxes. 


92.40 


7.60 


34 


91.50 


8.50 


26 


Communities should set their own standards, 
even if that means their schools are of uneven 
quality or lack the 'common school’ experience. 


33.80 


66.20 


94 


36.70 


63.30 


93 



During follow-up conversations/surveys, when we asked participants to indicate the extent to which they 
felt they had a strong understanding of standards-based education prior to participating in the 
conversation, the mean rating, participants rated themselves a 4.55. It is important to bear - this in mind 
when one examines (the mean rating) for the extent to which participants felt they left the conversation 
with an improved understanding of standards-based education. For this question, participants indicated 
that they did not feel that they learned that much (mean = 2.90). Flowever, as one participant noted, “I 
think that most people do not understand what 'standards-based' means. The dialogue was useful in 
making it more clear to some what it means.” 
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Although it is difficult to ascertain whether participants actually reached a shared understanding of 
standards-based education through their participation in the conversation, moderators suggested that 
alignment on some issues was evident — for some, this centered on the need for additional 
communication among schools, parents, and communities, for others shared understandings were around 
an agreed upon vision for education priorities. Other issues — notably the role of standards and the 
concomitant and requisite actions to enable and enforce standards — failed to meet the benchmark of a 
shared understanding. Arguably, however, the role and goal of the National Dialogue project is not to 
enforce or even strive for consensus-building around such broad issues as what role standards ought to 
play in educating youth; rather the role and goal of the Dialogue is simply to bring to the fore the 
disparate issues, thereby ensuring that different voices are given an opportunity to be heard. In other 
words, reaching a shared understanding is different from reaching agreement on priorities and next steps. 
One moderator summed this important sentiment up by suggesting that “everyone's opinion was heard 
and respected, but not necessarily agreed upon.” 

Question: Did the Dialogue facilitate the building of a bridge between community-level conversations 
and technical conversations? 

It is also important to determine whether the participants in the dialogue are comfortable engaging. 
Oftentimes, conversations on standards-based education and testing and assessment turn on very technical 
issues associated with benchmarks, alignment, or the psychometric properties of testing instruments. We 
were curious whether technical issues surface and if they did, who was engaged? In addition, we asked: 
To what extent are participants engaged with substantive ideas associated with SBE (e.g., defining 
standards, benchmarks, issues of assessment) in nontechnical ways ? 

During follow-up conversations and surveys, participants were asked to reflect on the extent to which 
they were able to understand the conversation, including any technical issues that arose. Participants 
indicated that they felt comfortable in their understanding, rating this item a 4.45. In addition, moderators 
noted that they did not feel as though technical details about standards or testing “took over” the 
conversations. Observations of the dialogues are also useful to consider in learning about whether 
substantive issues were addressed in ways that were not overly complicated. Over the past year, we’ve 
heard citizens engage in a number of dialogues on standards-based education, sometimes hinting at more 
technical issues, but seemingly using care in selecting their language so that the issues raised are 
comprehendible by all members of the dialogue. Thus, while the construct validity of a state test might 
have been questioned, the issues were raised using everyday terms. Likewise, standards-based 
conversations tended to focus on substantive issues about helping all children succeed and accountability 
conversations typically centered on how to ensure that accountability provisions are broadly understood 
and applied (e.g., to school personnel but also to parents, students, and sometimes community members). 

Question: Did the Dialogue facilitate a better understanding of other stakeholder perspectives? 

In terms of whether participants felt that they had a better understanding of fellow citizens’ perspectives 
following the conversation, follow-up conversations and surveys revealed that participants agreed (though 
not strongly) that they learned about each others’ perspectives (mean = 3.50). Others’ perspectives can 
sometimes be unexpected — encounters with such different, diverse perspectives have the potential to 
engender self-questioning and self-reflection. Thus, during follow-up with participants we asked whether 
there were any perspectives expressed during the Dialogue that surprised them. Although several 
participants indicated that they did not hear anything that they did not expect, the following comments arc 
indicative of the areas that provided new information or perspectives or that participants found 
particularly intriguing: 
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I am always grateful to the insight provided by the students. 

One woman said that it was ok to teach to the test, but we have to agree what the test 
was. That was food for thought. 

How much high school has changed since I attended. 

I was bothered to hear that some teachers did not like “high expectations’’ for their 
students — isn't that what we should all be striving fori 

I was amazed that some educators can be numb to the social/emotional needs of our 
students. ..we cannot look at children as merely test passers... 

People teaching continuation/alternative education, as well as our schools with high 
numbers of ELL student, have incredible challenges which the system is not significantly 
addressing. They MUST have more support to begin meeting the standards with their 
populations. 

Since I've been through a variation of meetings like this, I felt that I'cl heard much of it 
prior. It is always interesting to hear from the students. In our group they felt “over 
tested, ”, but also saw the benefits of having high standards. 

Moderators agreed that participants learned about each other’s viewpoints — and importantly, that 
participants learned about perspectives from those who are in different socioeconomic strata, different 
developmental periods as parents, and so forth than themselves. 

Overall Conclusions 

On the whole, the evaluation revealed that the National Dialogue met each of its overall goals. 
Participants and moderators reported high levels of engagement and open, respectful conversations that 
were meaningful and relevant. Moreover, demographic ally speaking, the conversations drew together 
diverse constituencies with varied perspectives, and although participants did not always reach consensus 
or completely understand one another’s perspectives, the exposure to varied understandings is considered 
crucial. Finally, participants did not report feeling disenfranchised because of technical issues — if 
anything, participants were deeply engaged in the ideas associated with standards-based education. 
Although actions and outcomes at the community level directly emanating from Dialogue participation 
are not discernable, arguably the Dialogue served its purposes well — particularly toward the end of 
getting the public engaged in the conversation and facilitating their discovery process in terms of their 
own perspectives regarding education and their fellow citizens’ perspectives regarding education. Table 7 
reiterates the goals of the National Dialogue, the evaluation questions, and our determination regarding 
the achievement of the goal. 



Table 7. Relationship between National Dialogue Goal, Evaluation Questions, and Data Sources 



National Dialogue goal: 


Evaluation questions: 


Goal 

Met? 


Allowed for meaningful exchanges of information 
through conversation 


Were participants engaged? 

Was an open and respectful 
conversation maintained? 

Was the conversation meaningful? 


✓ 
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National Dialogue goal: 


Evaluation questions: 


Goal 

Met? 


Enabled various levels of stakeholders to 
be engaged in conversations 


Were different constituencies involved in the 
dialogue? 


S 


Provided a useful, appropriate, timely 
conversation with sustained benefits 
(participants anticipate using the 
information) 


Was the conversation relevant (if so, in what ways)? 
Has the conversation led to proposed actions (if 
appropriate)? 


S 


Facilitated the development of a shared 
understanding of SBE 


Do participants have a shared understanding of SBE 
(where appropriate, have participants learned more 
about SBE)? 


s 


Facilitated the building of a bridge 
between community-level conversations 
and technical conversations 


Did technical issues surface? Who was engaged? 

To what extent arc participants engaged with 
substantive ideas associated with SBE (e.g., defining 
standards, benchmarks, issues of assessment) in non- 
technical ways? 




Facilitated a better understanding of 
other stakeholder perspectives 


To what extent do participants have a better 
understanding of fellow citizens’ perspectives? 


✓ 
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